Load Management and High Availability in the Borealis Distributed Stream Processing Engine
نویسندگان
چکیده
Borealis is a distributed stream processing engine that has been developed at Brandeis University, Brown University, and MIT. It extends the first generation of data stream processing systems with advanced capabilities such as distributed operation, scalability with timevarying load, high availability against failures, and dynamic data and query modifications. In this paper, we focus on aspects that are related to load management and high availability in Borealis. We describe our algorithms for balanced and resilient load distribution, scalable distributed load shedding, and cooperative and self-configuring high availability. We also present experimental results from our prototype implementation showing the effectiveness of these algorithms.
منابع مشابه
Fault-tolerance and load management in a distributed stream processing system
Advances in monitoring technology (e.g., sensors) and an increased demand for online information processing have given rise to a new class of applications that require continuous, lowlatency processing of large-volume data streams. These “stream processing applications” arise in many areas such as sensor-based environment monitoring, financial services, network monitoring, and military applicat...
متن کاملThe Design of the Borealis Stream Processing Engine
Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora [14] and distribution functionality from Medusa [51]. Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly requi...
متن کاملDesign Issues for Second Generation Stream Processing Engines
Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora [13] and distribution functionality from Medusa [49]. Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly requi...
متن کاملThe Design of the Borealis Stream Processing Engine
Borealis is a second-generation distributed stream processing engine that is being developed at Brandeis University, Brown University, and MIT. Borealis inherits core stream processing functionality from Aurora [11] and distribution functionality from Medusa [45]. Borealis modifies and extends both systems in non-trivial and critical ways to provide advanced capabilities that are commonly requi...
متن کاملof Informatics Design , Implementation , and Evaluation of Network Monitoring Tasks for the Borealis Stream Processing
Several data stream management systems (DSMSs), enabling their users to measure and analyze the behavior of a data stream on-line, are now available to the public. Network monitoring, as a challenging application domain, has shown to fit into the domain of DSMSs. With the use of the Borealis DSMS, we have implemented a set of network monitoring tasks. By performing these network monitoring task...
متن کامل